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Performance  of  External  Quality  Control  Reviews  by 
Military  Department  Audit  Agencies 


Executive  Summary 


Introduction.  The  Government  auditing  standards  issued  by  the  Comptroller  General 
of  the  United  States  require  organizations  conducting  audits  in  accordance  with 
Government  auditing  standards  to  undergo  an  external  quality  control  review  at  least 
once  every  3  years  by  an  organization  not  affiliated  with  the  organization  being 
reviewed.  The  external  quality  control  review  should  determine  whether  the 
organization’s  internal  quality  control  system  is  in  place  and  operating  effectively  to 
provide  reasonable  assurance  that  established  auditing  standards  were  followed.  Prior 
to  FY  1999,  the  Office  of  the  Inspector  General,  DoD,  conducted  external  quality 
control  reviews  of  the  Military  Department  audit  agencies.  However,  during  the 
September  25,  1998,  DoD  Audit  Chiefs’  Council  meeting,  it  was  agreed  that  the 
Military  Department  audit  agencies  would  conduct  external  quality  control  reviews  on 
each  other  in  FY  1999.  In  addition,  the  Council  agreed  that  the  agencies  would  use  the 
President’s  Council  on  Integrity  and  Efficiency  external  quality  control  review  guide  to 
perform  the  reviews. 

Objectives.  The  objectives  of  the  evaluation  were  to  determine  whether  the  Military 
Department  audit  agencies'  process  for  conducting  external  quality  control  reviews  of 
each  other,  as  established  by  the  DoD  Audit  Chiefs’  Council,  was  working  as  planned 
and  to  identify  the  types  of  findings  being  reported.  Specifically,  we  determined 
whether  reliance  could  be  placed  on  the  external  quality  control  review  results  to 
support  the  Inspector  General,  DoD,  responsibilities  for  oversight  of  Military 
Department  audit  agencies. 

Results.  The  Military  Department  audit  agencies  successfully  implemented  an  external 
quality  control  review  process.  The  results  of  the  FY  1999  reviews  indicated  that,  in 
each  case,  the  system  of  quality  control  was  designed  in  accordance  with  the  quality 
standards  established  by  the  President’s  Council  on  Integrity  and  Efficiency  and  each 
organization’s  internal  quality  control  system  was  operating  effectively.  We  can  rely 
on  the  external  quality  control  review  results  in  supporting  the  Office  of  the 
Inspector  General,  DoD,  oversight  responsibilities. 

Management  Comments.  We  provided  a  draft  of  this  report  on  August  14,  2000.  No 
written  response  was  required,  and  none  was  received.  Therefore,  we  are  publishing 
this  report  in  final  form. 
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Background 


Government  Auditing  Standards.  Government  auditing  standards  issued  by 
the  Comptroller  General  of  the  United  States  require  that  organizations 
conducting  Government  audits  have  an  appropriate  internal  quality  control 
system  in  place  and  undergo  an  external  quality  control  review.  Organizations 
conducting  audits  should  undergo  an  external  quality  control  review  at  least  once 
every  3  years  by  an  organization  not  affiliated  with  the  organization  being 
reviewed.  The  external  quality  control  review  should  determine  whether  the 
organization’s  internal  quality  control  system  is  in  place  and  operating 
effectively  to  provide  reasonable  assurance  that  established  auditing  standards 
were  followed.  An  external  quality  control  review  under  the  Government 
auditing  standards  should  meet  the  following  requirements.1 

•  Reviewers  should  be  qualified  and  have  current  knowledge  of  the 
type  of  work  to  be  reviewed  and  the  applicable  auditing  standards. 

•  Reviewers  should  be  independent  of  the  audit  organization  being 
reviewed,  its  staff,  and  its  auditees  whose  audits  are  selected  for 
review. 

•  Reviewers  should  use  sound  professional  judgment  in  conducting  and 
reporting  the  results  of  the  external  quality  control  review. 

•  Reviewers  should  select  audits  that  provide  a  reasonable  cross-section 
of  the  audits  conducted  in  accordance  with  the  Government  auditing 
standards  or  select  audits  that  provide  a  reasonable  cross-section  of 
the  organization’s  audits,  including  one  or  more  audits  conducted  in 
accordance  with  Government  auditing  standards. 

•  The  review  should  include  a  review  of  the  audit  reports,  working 
papers,  and  other  necessary  documents,  as  well  as  interviews  with 
the  reviewed  organization’s  professional  staff. 

•  A  written  report  should  be  prepared  communicating  the  results  of  the 
external  quality  control  review. 

•  External  quality  control  review  procedures  should  be  tailored  to  the 
size  and  nature  of  an  organization’s  audit  work. 

President’s  Council  on  Integrity  and  Efficiency  Review  Guide.  The 

President’s  Council  on  Integrity  and  Efficiency  (PCIE),  “Guide  for  Conducting 
External  Quality  Control  Reviews  of  the  Audit  Operations  of  Offices  of 
Inspector  General,”  April  1997,  (Review  Guide)  presents  the  standards  and 
detailed  guidance  for  conducting  external  quality  control  reviews  of  the  audit 
operations  of  audit  organizations.  The  Review  Guide  was  developed  to  promote 
consistency  in  conducting  the  reviews.  An  external  review  requires 


1  External  quality  control  reviews  conducted  through  or  by  the  President's  Council  on  Integrity  and 
Efficiency  meet  these  requirements. 
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considerable  professional  judgment  on  the  part  of  the  review  team.  The 
guidance  contained  in  the  PCIE  Review  Guide  is  advisory  in  nature  and  is  not 
intended  to  supplant  the  review  team’s  professional  judgment  on  the  approach  to 
take  or  the  specific  procedures  to  be  performed  in  specific  reviews.  The  PCIE 
external  quality  control  review  program  is  intended  to  be  positive  and 
constructive,  rather  than  negative  or  punitive. 

DoD  Audit  Chiefs’  Council.  The  DoD  Audit  Chief's  Council  is  composed  of 
the  heads  of  the  DoD  audit  organizations.  The  organizations  include  the  office 
of  the  Assistant  Inspector  General  for  Auditing,  DoD;  the  Defense  Contract 
Audit  Agency;  the  Army  Audit  Agency  (AAA);  the  Naval  Audit  Service  (NAS); 
and  the  Air  Force  Audit  Agency  (AFAA).  The  council  meets  periodically  under 
the  leadership  of  the  Inspector  General,  DoD,  to  discuss  audit  issues  and  matters 
that  affect  the  DoD  audit  community.  The  September  25,  1998,  DoD  Audit 
Chiefs’  Council  meeting  established  that  Military  Department  audit  agencies 
would  conduct  external  quality  control  reviews  on  each  other  in  FY  1999.  In 
addition,  the  Council  agreed  that  the  agencies  would  use  the  PCIE  Review 
Guide  to  perform  external  quality  control  reviews. 

Review  Schedule.  The  Military  Department  audit  agencies  agreed  on  the 
following  scheduling  option  for  FY  1999. 

•  The  AAA  would  review  the  NAS 

•  The  NAS  would  review  the  AFAA. 

•  The  AFAA  would  review  the  AAA. 

Objectives 


The  objectives  of  the  evaluation  were  to  determine  whether  Military  Department 
audit  agencies'  process  for  conducting  external  quality  control  reviews  of  each 
other,  as  established  by  the  DoD  Audit  Chiefs’  Council,  was  working  as 
planned  and  to  identify  the  types  of  findings  being  reported.  Specifically,  we 
determined  whether  reliance  could  be  placed  on  die  external  quality  control 
review  results  to  support  the  Inspector  General,  DoD,  responsibilities  for 
oversight  of  Military  Department  audit  agencies. 
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External  Quality  Control  Reviews 

The  Military  Department  audit  agencies  successfully  implemented  an 
external  quality  control  review  process.  Using  the  PCIE  Review  Guide, 
review  teams  completed  assessments  of  the  AAA,  NAS,  and  AFAA. 

The  results  of  the  FY  1999  reviews  indicated  that,  in  each  case,  the 
system  of  quality  control  was  designed  in  accordance  with  the  quality 
standards  established  by  the  PCIE  and  that  each  organization’s  internal 
quality  control  system  was  operating  effectively.  We  can  rely  on  the 
audit  agencies'  external  quality  control  review  results  to  support  the 
Office  of  the  Inspector  General,  DoD,  oversight  responsibilities. 

Quality  Control  Review  Process 

The  audit  agencies  successfully  implemented  an  external  quality  control  review 
process.  Using  the  PCIE  Review  Guide,  review  teams  completed  assessments 
of  the  AAA,  NAS,  and  AFAA.  In  addition,  discussions  with  personnel 
involved  indicated  that,  overall,  the  process  worked  well.  However, 
Government  auditing  standards  do  not  allow  the  audit  agencies  to  rotate  external 
quality  control  reviews  among  each  other. 

PCIE  Review  Guide  Checklists,  The  audit  agencies  used  the  PCIE  Review 
Guide  for  the  external  quality  control  review  as  agreed  to  at  the 
September  25,  1998,  DoD  Audit  Chiefs’  Council  meeting.  The  review 
approach  advocated  by  the  PCIE  Review  Guide  is  to: 

•  gain  an  understanding  of  the  reviewed  organization’s  audit  operation 
and  its  internal  quality  control  system; 

•  gain  an  understanding  of  the  organization’s  internal  quality  control 
program,  including  a  review  of  any  internal  reports; 

•  use  the  knowledge  obtained  from  the  preceding  steps; 

•  select  the  offices  and  audits  to  be  reviewed  and  the  nature  and  extent 
of  the  tests  to  perform;  and 

•  review  functional  areas  and  individual  audits. 

To  meet  the  objectives  described  above,  the  PCIE  Review  Guide  contains 
several  checklists  as  guidance,  as  follows: 

•  Appendix  A  is  a  questionnaire  designed  to  obtain  general  information 
about  an  audit  organization  and  its  internal  quality  control  system. 

•  Appendix  B  contains  guidance  to  help  document  the  review  team’s 
understanding  and  to  evaluate  the  adequacy  and  quality  of  the  audit 
organization’s  policies  and  procedures. 
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•  Appendix  C  contains  guidance  on  assessing  the  design  of  the  internal 
quality  control  program. 

•  Appendix  D  contains  general  guidance  on  reviewing  internal  quality 
control  reports. 

•  Appendix  F  is  a  checklist  for  reviewing  individual  performance 
audits. 

Augmentation  of  Checklists.  To  assist  in  their  reviews,  the  audit  agencies’ 
review  teams  augmented  the  checklists  with  additional  steps  and  references  from 
each  audit  agency's  policies  and  procedures.  For  example,  for  the  AAA 
external  quality  control  review  of  the  NAS,  the  AAA  added  steps  to  the  PCIE 
Review  Guide  checklist  for  individual  NAS  performance  audits.  Those  steps 
were  added  to  the  review  of  NAS  quality  control  and  asked  questions  such  as 
whether  the  draft  report  was  referenced  prior  to  issuance  and  whether  the 
referencer  was  independent.  In  addition,  specific  citations  on  quality  control 
from  the  Naval  Audit  Handbook  were  included  in  the  checklist. 

Satisfaction  with  the  Process.  Discussions  with  personnel  from  the  audit 
agencies  revealed  that  the  external  quality  control  review  process  worked  well. 
We  held  discussions  with  13  personnel  from  the  audit  agencies  who  were 
involved  in  conducting  the  reviews,  and  they  expressed  overall  satisfaction  with 
the  external  quality  control  review  process.  Items  that  personnel  stated  worked 
particularly  well  included  the  following. 

•  The  memorandums  of  understanding  between  audit  agencies  provided 
the  rules  for  the  external  quality  control  reviews. 

•  Pre-planning  meetings  between  the  audit  agencies  also  assisted  in 
establishing  the  rules  for  the  review.  Conducting  presentations  on 
using  the  PCIE  Review  Guide  and  experiences  from  prior  external 
quality  control  reviews  also  helped. 

•  Obtaining  as  much  information  in  advance  of  the  reviews  enabled  the 
review  teams  to  incorporate  specific  sentences  and  references  from 
the  reviewed  agencies'  policies  and  procedures  into  the  PCIE  Review 
Guide  checklists. 

•  Using  point  papers  during  the  review.  At  on-site  exit  conferences, 
the  visiting  auditors  provided  the  site  manager  with  a  point  paper  that 
summarized  all  the  issues. 

•  The  external  quality  control  reviews  enabled  the  personnel  involved 
to  gain  an  understanding  of  another  audit  organization’s  methodology 
and  how  other  personnel  were  working. 

Rotation  of  Assignments.  The  Government  auditing  standards  do  not  allow 
audit  agencies  to  rotate  external  quality  control  reviews  among  each  other 
because  an  audit  organization  is  not  permitted  to  review  the  organization  that 
conducted  its  most  recent  external  quality  control  review.  The  Government 
auditing  standards  requirement  is  intended  to  ensure  that  independence  is 
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maintained  both  in  appearance  and  fact.  With  only  three  organizations 
involved,  assignments  cannot  be  rotated.  For  example,  the  NAS  cannot  review 
the  AAA  for  the  FY  2002  external  quality  control  review  because  the  AAA 
reviewed  the  NAS  for  the  FY  1999  review.  As  such,  the  NAS  would  have  to 
review  the  AFAA  again  in  FY  2002.  We  agree  that  there  was  a  conflict 
between  the  Government  auditing  standard  and  the  external  quality  control 
review  plan.  However,  the  DoD  Audit  Chiefs  can  reassess  the  external  quality 
control  review  plan  before  the  scheduled  FY  2002  reviews  and  change  the  plan 
if  necessary. 

Types  of  Findings  Reported 


For  each  of  the  three  reviews,  the  review  teams  reported  that  the  quality  control 
system  was  designed  in  accordance  with  the  quality  standards  established  by  the 
PCIE.  Further,  the  review  teams  reported  that  each  organization’s  internal 
quality  control  system  was  operating  effectively  to  provide  reasonable  assurance 
that  audit  personnel  were  following  established  policies,  procedures,  and 
applicable  auditing  standards. 

The  review  teams  also  made  observations  and  suggested  actions  to  the  audit 
agencies  on  certain  areas  to  strengthen  their  systems  of  quality  control. 
Generally,  the  audit  agencies  took  action  to  implement  the  teams'  review 
suggestions,  except  that  the  AFAA  had  not  established  a  formal  system  to  track 
all  open  recommendations  from  internal  quality  control  reviews.  AFAA 
Instruction  65-105,  “Internal  Quality  Control  Review  Program,”  January  14, 
2000,  requires  the  establishment  of  such  a  system.  However,  the  AFAA 
completed  action  to  establish  a  follow-up  system  during  our  review. 

Reliance  on  External  Quality  Control  Reviews 


We  can  rely  on  the  audit  agencies'  external  quality  control  review  results  in 
supporting  the  Office  of  the  Inspector  General,  DoD,  oversight  responsibilities. 
Working  papers  documented  the  work  performed  and  conclusions  reached,  and 
there  were  no  indications  that  any  significant  deficiencies  were  found  but  not 
reported.  In  addition,  the  review  teams  used  the  PCIE  Review  Guide  checklists. 
Also,  memorandums  of  understanding  were  prepared,  there  were  no  indications 
of  any  impairment  to  independence,  experienced  audit  staff  were  assigned  to  the 
review,  and  a  reasonable  cross-section  of  audit  reports  was  chosen  to  review. 

Memorandums  of  Understanding.  The  audit  agencies  prepared  memorandums 
of  understanding  to  establish  the  framework  for  the  reviews.  Items  outlined  in 
the  memorandums  were  the  contact  persons,  the  review  schedule  and 
coordination,  the  scope  of  the  review,  the  results  report  process,  and  retention 
of  and  access  to  the  working  papers. 

Working  Papers  and  Reports.  Each  of  the  review  teams  prepared  working 
papers  to  document  the  work  performed  and  the  conclusions  reached  for  the 
review,  and  there  was  no  indication  in  the  working  papers  that  any  significant 
deficiencies  were  found  but  not  reported.  In  addition,  report  conclusions  were 
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supported  by  and  referenced  to  the  working  papers.  We  did  not  retest  any  of 
the  results  because  the  review  of  the  working  papers  allowed  us  to  place  reliance 
on  the  review  results. 

Impairment  to  Independence.  There  were  no  indications  that  the  audit 
agencies  had  any  impairments  to  independence.  According  to  the  PCIE  Review 
Guide,  the  review  team  members  should  meet  the  independence  standards  of  the 
Government  auditing  standards.  In  addition,  former  employees  employed 
within  the  past  2  years  of  the  organization  being  reviewed  should  generally  not 
be  selected  to  be  part  of  the  review  team.  Based  on  the  impairments  listed  in 
the  Government  auditing  standards,  nothing  came  to  our  attention  to  indicate 
that  the  agencies  had  any  impairments  to  independence.  In  addition,  none  of  the 
agency  review  team  members  had  been  employed  by  the  organization  being 
reviewed  for  the  last  2  years. 

Experience  of  Review  Staff.  An  experienced  audit  staff  was  assigned  by  the 
Military  Department  audit  agencies  to  conduct  the  external  quality  control 
reviews.  Each  review  team  had  seven  to  eight  personnel  who  were  mainly  high 
graded  operational  auditors  with  many  years  of  experience.  For  example,  two 
members  of  the  NAS  review  team  had  27  and  20  years  of  auditing  experience, 
respectively. 

Reasonableness  of  Reports  Selected.  The  Military  Department  audit  agencies 
selected  a  reasonable  cross-section  of  audit  reports  to  review.  Each  review  team 
selected  six  to  eight  audit  reports  to  review  as  well  as  one  to  two  internal  quality 
control  reports  to  review.  The  following  criteria  were  used  to  select  the  audit 
reports  to  review. 

•  The  location  and  size  of  the  audit  offices.  Larger  offices  where 
working  papers  would  be  stored  and  numerous  audits  conducted  were 
selected. 

•  The  number  of  findings  and  recommendations.  Audit  projects  with 
more  significant  results  were  selected. 

•  Audit  identified  potential  monetary  benefits.  Audits  that  identified 
areas  where  funds  could  be  put  to  other  uses  were  selected. 

•  The  amount  of  time  charged  to  the  project.  Larger,  more  significant 
audits  were  chosen. 

Conclusion 


Overall,  the  Military  Department  audit  agencies’  external  quality  control 
reviews  during  FY  1999  worked  well.  As  such,  we  have  determined  that  we 
can  rely  on  the  results  of  the  reviews  to  support  our  oversight  responsibilities. 
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Appendix  A.  Evaluation  Process 

Scope 


We  based  our  review  on  the  Government  auditing  standards  relating  to  quality 
control,  the  PCIE  Review  Guide  for  external  quality  controls,  and  DoD  policies 
and  procedures  in  force  from  1991  through  1999.  In  addition,  we  reviewed 
FY  1999  external  quality  control  review  reports  of  the  Military  Department 
audit  agencies,  pre-planning  meeting  notes,  and  memorandums  of 
understanding.  Each  audit  agency  review  team  selected  six  to  eight  audit 
reports  and  one  to  two  internal  quality  control  reports  to  review.  We  reviewed 
all  working  papers  supporting  those  reviews.  We  also  discussed  the  external 
quality  control  review  process  with  various  personnel  involved  to  determine 
whether  there  were  any  problems  associated  with  or  ways  to  improve  the 
process. 

Methodology 


To  identify  the  types  of  external  quality  control  review  findings  being  reported, 
we  obtained  the  FY  1999  Military  Department  audit  agencies'  external  quality 
control  review  reports.  To  determine  whether  the  process  for  conducting 
external  quality  control  reviews  was  working  as  planned  and  whether  reliance 
could  be  placed  on  the  results,  we  discussed  the  process  with  various  personnel 
and  reviewed  the  audit  agencies'  working  papers  to  determine  whether  the  PCIE 
Review  Guide  was  followed  and  whether  any  significant  deficiencies  were  found 
but  not  reported.  We  did  not  retest  the  Military  Department  audit  agencies' 
work  performed  for  the  external  quality  control  review.  In  addition,  we 
followed  up  with  the  audit  agencies  to  determine  whether  the  suggestions  made 
by  the  review  teams  to  improve  internal  quality  control  systems  were 
implemented. 

Use  Of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  evaluation. 

Evaluation  Dates  and  Standards.  We  performed  this  evaluation  from 
February  through  July  2000,  in  accordance  with  standards  issued  and 
implemented  by  the  Inspector  General,  DoD. 

Management  Control  Program  Review.  The  management  controls  over  the 
Military  Department  audit  agencies'  process  for  conducting  external  quality 
control  reviews  of  each  other  were  adequate.  We  identified  no  material 
management  control  weaknesses.  One  of  the  methods  an  organization  uses  to 
determine  whether  it  has  any  management  control  deficiencies  is  the 
consideration  of  inspections  and  other  independent  review  reports.  For  the 
FY  1999  Statements  of  Assurance  on  Management  Controls,  the  Military 
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Department  audit  agencies  considered  the  results  of  the  external  quality  control 
reviews  when  evaluating  the  adequacy  and  effectiveness  of  their  management 
controls. 

Contacts  During  the  Evaluation.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Further  details  are  available  upon  request. 

Prior  Coverage 

No  prior  coverage  has  been  conducted  on  the  subject  during  the  last  5  years. 
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Appendix  B.  Report  Distribution 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 

Department  of  the  Navy 

Auditor  General,  Department  of  the  Navy 

Department  of  the  Air  Force 

Auditor  General,  Department  of  the  Air  Force 

Non-Defense  Federal  Organization 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 
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Evaluation  Team  Members 


The  Deputy  Assistant  Inspector  General  for  Audit  Policy  and  Oversight,  Office 
of  the  Assistant  Inspector  General  for  Auditing,  DoD,  prepared  this  report. 

Patricia  A.  Brannin 
Wayne  C.  Berry 
Martin  T.  Heacock 
Robert  L.  Kienitz 
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